Approval for this study was obtained from our Institutional Review Board. The raw dataset consisted of 62,801 Press Ganey® Medical Practice surveys from 34,534 patients receiving care from 664 providers between 1/1/2013 and 12/31/2013 from our institution, which is a University-based health care system comprised of hospitals and clinics that provide primary through tertiary care for over 200 medical specialties. Patients were sent an email with a link to the Press Ganey® survey following their visit. Surveys missing all 24 items were excluded. If a patient filled out multiple surveys in 2013, only the first survey with at least 1 of the 24 items answered was kept. The final analysis data set contained 34,503 surveys from unique patients seen by 624 providers from 47 specialties and 94 clinics.

The Press Ganey® survey has 24 items organized into 6 scales: Access (4 items), Moving Through the Visit (2), Nurse Assistant (2), Care Provider (10), Personal Issues (4) and Overall Assessment (2). Each item was scored as follows: very poor (score = 0), poor (25), fair (50), good (75) and very good (100). The score for each scale was calculated from the mean scores of all items within the scale, and the mean total score was calculated from the mean scores from the six scales weighted equally. The Press Ganey® survey scoring instructions document is available by contacting Press Ganey® [15]. Consistent with factor analyses performed on the CAHPS surveys, confirmatory factor analysis was performed on the first 22 questions composing the first five scales [7–9, 11, 12], as the sixth scale corresponded to Overall Assessment (two questions), which correlated with the other scales.

We sought to evaluate both the reliability and validity of the Press Ganey® survey to assess its utility for measuring patient satisfaction. Reliability is the extent to which a survey measures true signal. Validity indicates the extent to which a survey measures what it is intended to measure. In particular we evaluated the following Press Ganey® survey properties: 1) data quality, 2) internal consistency reliability of items within each scale, 3) factor structure, 4) convergent validity and 5) discriminant validity.

Items, scales and total scores were evaluated for missingness, the percentage of values hitting the floor (minimum value) and ceiling (maximum value), and skewness. To assess whether items were missing completely at random, we used Little’s MCAR test implemented in the BaylorEdPsych package in R [16]. High floor and ceiling rates yield reduced power to discriminate among patients who have low or high satisfaction, respectively. Floor and ceiling rates were defined as rare if they occurred 20% of the time [17, 18]. These data quality metrics can impact the reliability and validity of a survey [17, 18]. In particular, substantial ceiling (or floor) rates can notably impact percentile rankings of scores within an institution. To investigate this we converted the raw scores to percentile ranks using two different methods: method 1) the empirical cumulative distribution function in R ecdf(), and method 2) dividing the rank of a score by the number of scores. To assess how a provider’s percentile rank score could change quarterly, we calculated percentile rank scores within each quarter, and then averaged the raw and percentile rank scores for each provider by quarter. We then summarized the median change in consecutive quarters for raw and percentile rank scores among a) all providers, and b) providers who had a perfect score in at least one quarter.

We analyzed internal consistency reliability and homogeneity of items within each scale using Cronbach’s alpha and inter-item correlations [19, 20]. Nunnally recommends a minimum of 0.7 for Cronbach’s alpha [21]. Briggs and Cheek [22] suggested mean interitem correlations in the range of 0.1–0.5, and Clark and Watson [23] encouraged all interitem correlations to fall within this range. Smaller values (0.5) suggest item redundancy. We also calculated item-scale correlations, corrected for item overlap, where a 0.4 benchmark supported internal consistency [21]. The goal of these analyses was to verify consistency of items within scales, which is important for yielding reproducible results.

We explored the factor structure of the Press Ganey® survey numerically using confirmatory factor analysis (CFA) and visually using multi-dimensional scaling (MDS). The MDS plot provided a tool to visualize how items clustered within scales and inter-relationships among the scales based on Euclidean distances between the items. CFA was used to test whether the data fit the Press Ganey® survey design using the fit indices: root mean square error of approximation (RMSEA, cut off for good fit 0.95), Tucker-Lewis index (TLI, >0.9), and standardized root mean square residual (SRMR,